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Background and Objectives: Medical insurance claims (MIC) data are one of the largest sources of outcome data in the form of Interna- 
tional Classification of Diseases (ICD) codes. We evaluated the validity of the ICD codes from the Korean National MIC data with respect to 
the outcomes from acute myocardial infarction (AMI) in the Korean Heart Study. 

Subjects and Methods: Baseline information was obtained from health examinations conducted from 1994 to 2001. Outcome informa- 
tion regarding the incidence of AMI came from hospital admission discharge records from 1994 to 2007. Structured questionnaires were 
sent to 98 hospitals. In total, 107 cases of AMI with ICD codes of 121- (93 men, 26-73 years of age) were included in the final analyses. ICD 
code accuracy and reliability (kappa) for AMI were calculated. 

Results: A large number of AMI cases were from hospitals located in the Seoul area (75.9%). The accuracy of AMI was 71.4%, according 
to World Health Organization criteria (1997-2000, n=24, kappa=0.46) and 73.1% according to the European Society of Cardiology/American 
College of Cardiology (ESC/ACC) criteria (2001-2007, n=83, kappa=0.74). An age of 50 years or older was the only factor related to inaccuracy 
of codes for AMI (odds ratio, 4.6; 95% confidence interval, 1.2-17.7) in patients diagnosed since January 2001 using ESC/ACC criteria (n=83). 
Conclusion: The accuracy for diagnosing AMI using the ICD-10 codes in Korean MIC data was >70%, and reliability was fair to good; how- 
ever, more attention is required for recoding ICD codes in older patients. (Korean Circ J 201 2;42:10-1 5) 
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Introduction 

The importance of valid data has increased due to the high impact 
of cardiovascular diseases on mortality and morbidity. Medical in- 
surance claims (MIC) data are one of the largest sources of outcome 
data in the form of International Classification of Diseases (ICD) 
codes. 1 ' Because enrollment is mandatory for all Koreans, the Kore- 
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an national health insurance system covers nearly all of the coun- 
try's population (96.3% in 2007) with the exception of some medi- 
cal assistance beneficiaries. If the validity of MIC data is acceptable 
for research purposes, it can be used as a comprehensive and eco- 
nomical source of outcome information. 

International Classification of Diseases codes have been used in 
research for years; however, accuracy issues have been raised in 
recent studies. Questions regarding validity have been suggested 
as a result of ambiguities and inconsistencies in ICD-coded diag- 
noses. 2 ' Coding errors are also regarded as an issue with regards to 
data reliability. 3 ' Despite these concerns, the reported accuracy of 
ICD codes appeared to be acceptable in previous reports. In Korea, 
the accuracy rate of ICD codes for cerebrovascular disease was 83% 
for 425 men, according to insurance claims. 4 ' However, the accuracy 
of ICD codes for cardiovascular disease is not well-known. Particu- 
larly, acute myocardial infarction (AMI), a fatal disease, has doubled in 
the past 11 years (1997-2007). 5 ' Therefore, in this study we evaluat- 
ed the validity of ICD codes for AMI according to Korean national 
MIC data. 
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Subjects and Methods 
Study sample 

The subjects were participants in the prospective cohort Korean 
Heart Study, which was developed to establish a cardiovascular dis- 
eases prediction model. The initial Korean Heart Study population 
consisted of 476529 subjects who underwent health examinations 
at 19 health examination centers in Korea from 1993 to 2004. Am- 
ong the initial subjects, 673 cases with International Classification 
of Diseases 10th Revision codes (ICD-10) of cardiovascular disease 
outcomes (I20-I25, 1 50) on MIC were selected. Among the 482 re- 
sponses, 107 cases of AMI with the ICD codes of 121- (121, 1210-1219) 
were included in the final analyses (93 men, 26-73 years of age). Data 
from three cases could not be collected due to hospital closures and/ 
or mistakes in patient identification. 

The outcome variable investigated was incidence of AMI in hos- 
pital admission discharge records from 1994 to 2007 (median follow- 
up duration, 9.4 years). Outcomes were ascertained from health in- 
surance claim data from the National Health Insurance Corporation. 6 ' 71 

Informed consent from each study participant was obtained dur- 
ing routine health examinations. This study was approved by the In- 
stitutional Review Board of Human Research of University. 

Data collection 

Diagnosis determination (definition) 

The ICD-10 code for AMI was 121 -. The event validation committee 
of the Korean Society of Cardiology (KSC) also defined the diagnos- 
tic criteria for AMI according to the joint European Society of Cardio- 
logy/American College of Cardiology (ESC/ACC) criteria of 20 00 3 ' 
and World Health Organization (WHO) criteria. 9 ' Because the ESC/ 
ACC criteria were introduced in 2000, AMI was defined by WHO cri- 
teria until December 2000 (24 cases) and by ESC/ACC criteria since 
January 2001 (83 cases). The cut-off date was determined as above 
due to the possibility of a time lag between the publishing point of 
the new criteria and awareness of the new criteria to the majority 
of physicians. 

Questionnaire 

The questionnaire and survey were performed in cooperation with 
the KSC to improve the quality and rate of responses. To investi- 
gate the cardiovascular disease evidence reported in medical re- 
cords, a structured questionnaire was drafted by a cardiologist us- 
ing well-known diagnostic criteria by reviewing previous reports. 8 " 10 ' 
The event validation committee of the KSC, consisting of 23 cardi- 
ologists in teaching hospitals, reviewed the questionnaire. After in- 
put from the committee, two cardiologists and a family physician 
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tested and updated the questionnaire through a pilot study at one 
hospital with 20 cases. 

The final revised questionnaire was sent directly to the cardiology 
departments of 98 hospitals, which made the diagnosis using the 
ICD codes. If an individual cardiologist involved in the case had left 
the hospital, other staff or cardiologists were requested to answer 
the questionnaire. They were required to complete the questionnaire 
according to the medical records kept at their hospitals. A detailed 
introduction to the study purpose and guidelines for completing the 
questionnaire were provided. 

Statistical analysis 

Demographic and hospital characteristic data, including the num- 
ber of hospitals, were collected and analyzed according to area, ini- 
tial cases, returned cases, return rate, hospital type, questionnaire 
respondent, and ICD code administrator. The proportion of ICD codes 
per subject was calculated. 

The accuracy of ICD codes for AMI was calculated using the fol- 
lowing formula 4 ': the number of Ml patients according to the diag- 
nostic criteria/the number of patients with ICD-10 codes of 121 - in 
the MIC data among patients with sufficient medical records data 
x100 (%). 

Kappa values were calculated to assess the reliability between 
each of the AMI criteria and ICD codes. 

Potential factors related to the inaccuracy of an AMI diagnosis 
were assessed by logistic regression analysis. 

All analyses were conducted using SAS statistical software version 
9.1 (SAS Institute Inc., Cary, NC, USA). All statistical tests were two- 
sided, and statistical significance was determined to be p<0.05. 

Results 

Among the 673 mailed questionnaires, we received 482 respons- 
es (71.6%). The most common respondents were resident trainees 
in internal medicine departments (Table 1). Among the respondents, 
95.0°/o were working at hospitals affiliated with medical colleges. A 
large number of the cases (75.9%) were from hospitals located in 
the Seoul area (Table 2). AMI, defined as 121, 1210-1219 on the MIC 
ICD-10 codes, was reported in 107 cases (93 men, 26-73 years of 
age). Mean age (standard deviation) of the patients with AMI was 
49.5 (10.1) years. Unspecified forms of AMI (1219) accounted for 
48.6% (Table 3). The accuracy of AMI was 71.4%, according to the 
WHO criteria (1997-2000, n=24, kappa=0.46) and 73.1% according 
to ESC/ACC criteria (2001-2007, n=83, kappa=0.74), after excluding 
unavailable cases (Table 4 and 5). 

An age of 50 years or older was the only factor related to AMI in- 
accuracy (odds ratio, 4.6; 95% confidence interval, 1.2-17.7, p= 
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0.0270) after adjusting for gender, hospital location, respondents, 


Discussion 






ICD code recorder, and respondents who made the diagnosis relat- 








ed to the ICD codes (Table 6). 




The accuracy and reliability of AMI diagnoses in the Korean Heart 


Table 1. Characteristics of the respondents in the event validation project 


Table 2. Number of cases based on the characteristics of the hospitals par- 


N 


% 


ticipating in the event validation project (n=482) 




Questionnaire respondent 




N 


o/6 


Cardiologist 81 


26.3 


Type of hospitals 


Physician 25 8.1 


Hospital affiliated with medical college 458 


95.0 


Resident in internal medicine 192 


62.3 


Public hospital 


14 


2.9 


Resident in other department 2 


0.6 


UCMCIdl MLoUlldl Ul LIIIIIL 


Q 
O 


1.7 


Nurse 8 


2.6 


Unknown 


2 


0.4 


No response 182 




Region of hospital 






ICD code administrator 




Seoul 


366 


75.9 


Cardiologist 67 


22.9 


Gyeonggi 


97 


20.1 


Resident in internal medicine 199 


68.2 


Gangwon 


8 


1.7 


Registered medical record administrator 26 


8.9 


Gyeongnam 


3 


0.6 


No response 198 


Jeonbuk 


3 


0.6 


"Did you input the ICD code of the patient 




Gyeongbuk 


1 


0.2 


identified in this questionnaire personally?" 




Jeonnam 


1 


0.2 


Yes 52 


16.9 


Chungnam 


1 


0.2 


No 256 


83.1 


Chungbuk 


0 


0 


No response 182 


Jeju 


0 


0 


ICD: International Classification of Diseases 




Unknown 


2 


0.4 


Table 3. Subgroup of acute myocardial infarction (121 -) based on ICD-10 code frequency 


ICD-10 Diagnosis 




Men (n) 


Women (n) 


Total, n (o/o) 


121 Acute Ml 




6 


0 


6 (5.6) 


1210 Acute transmural Ml of anterior wall 




16 


3 


19(17.8) 


1211 Acute transmural Ml of inferior wall 




17 


2 


19(17.8) 


1212 Acute transmural Ml of other sites 




4 


1 


5 (4.7) 


1213 Acute transmural Ml of unspecified site 




1 


0 


1 (0.9) 


1214 Acute subendocardial Ml 




3 


2 


5 (4.7) 


1219 Acute Ml, unspecified 




46 


6 


52 (48.6) 


Total 121- Acute Ml 




93 


14 


107 


Ml: myocardial Infarction, ICD: International Classification of Diseases 










Table 4. Accuracy of ICD codes for acute myocardial infarction using the WHO criteria in medical insurance claims data among ICD-10 codes 121 


-(1997-2000) 


Medical records 


Diagnosis 


N 


(o/o) 


Availability Interpretation 


Available Possible 


Accurate 


15 


71.4 




Inaccurate 


6 


28.6 




Subtotal 


21 


100.0 


Impossible* 






3 


12.5 


Unavailable* 






0 




Total 






24 




Kappa=0.46 



*Due to insufficient data to determine the accuracy of ICD coding for AMI, + Due to closing of hospitals, failure to find medical records, or patient misiden- 
tification. ICD: International Classification of Diseases 
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Table 5. Accuracy of ICD codes for acute myocardial infarction using ESC/ACC criteria for medical insurance claim data with an ICD-10 code of 121- (2001-2007) 





Medical records 


Diagnosis 


N 


(%>) 


Availability 


Interpretation 


Available 


Possible 


Accurate 


57 


73.1 






Inaccurate 


21 


26.9 






Subtotal 


78 


100.0 




Impossible* 




3 


3.7 


Unavailable 1 " 






2 




Total 






83 




Kappa=0.74 



*Due to insufficient data to determine the accuracy of ICD coding for AMI, f Due to the closing of hospitals, failure to find medical records, or patient mis- 
identification. ICD: International Classification of Diseases, ESC/ACC: European Society of Cardiology/American College of Cardiology 

Table G. Potential variables related to the inaccuracy of ICD codes for acute myocardial infarction among patients diagnosed since January 2001 by ESC/ 
ACC criteria (n=83) 



Variables 


Category 


Odds ratio (95% CI) 


P 


Age 


<50 


1.0 






>50 


4.6(1.2-17.7) 


0.0270 


Sex 


Men 


1.0 






Women 


1.6 (0.2-16.1) 


0.7101 


Location 


Seoul 


1.0 






Other area 


4.3 (0.1-129.8) 


0.3990 


Questionnaire respondents 


Cardiologist 


1.0 






Others 


12.0(0.7-216.1) 


0.0927 


ICD code recorder 


Cardiologist 


1.0 






Others 


0.9 (0.1-13.4) 


0.9131 


Respondent made the diagnosis of the ICD codes 


Yes 


1.0 






No 


1.7 (0.8-3.6) 


0.1579 



CI: confidence interval, ICD: International Classification of Diseases, ESC/ACC: European Society of Cardiology/American College of Cardiology 



Study was evaluated to assess validity. The accuracy of AMI was 71.4% 
(WHO criteria, 1997-2000) to 73.1% (ESC/ACC criteria, 2001-2007). 
Reliability between each of the AMI criteria and ICD codes was good 
to fair in terms of the kappa values (0.46 or 0.74) according to the 
suggestion of Fleiss that values >0.75 indicate excellent reliability, 
those 0.40-0.75 indicate fair to good, and <0.40 indicate poor. 11 ' 

The importance of MIC data has been growing, particularly in 
epidemiological studies due to the huge population included. 121 How- 
ever, issues regarding the reliability of MIC data have been raised 
continuously. Hospitals may record codes incorrectly in some cases 
due to reimbursement issues. 131 Admission duration and the pres- 
ence of complex co-morbidities also add to confusion in underst- 
anding the discharge data. 14)151 There are several known reasons as- 
sociated with coding errors, including misspecification, miscoding, 
and re-sequencing or substitution of the major diagnosis with a se- 
condary diagnosis for the purpose of reimbursement. 161 The ICD 
codes alone include ambiguities and inconsistencies. 2 ' 

Although the ICD-coding system has certain limitations, the use- 



fulness of the coding has also been reported, and it has been used 
in various studies to date. In an intervention study, the reliability of 
ICD-10 coding was assessed among three groups of students, staff, 
and coding specialists, and concordance results were fair to mod- 



erate 



17) 



Studies on the reliability of ICD codes in the field of cardiovascu- 
lar research have been reported. Validity of ischemic heart disease 
coding was evaluated in Canada where physician service claims and 
discharge records using the ICD-9 code were concordant in 69.3% 
of AMI cases. 181 

In the Republic of Korea, the National Insurance System covers 
nearly all of the population. Claims and payments are handled by an 
online electronic system. Some conflicts occur with the payment 
system between the insurance payer and hospitals, so we could 
not exclude the possibility of systematic miscoding; however, ef- 
forts to reduce coding errors have been introduced by the payer. 

Several findings regarding stroke and diabetes have been obtained 
among reports on the accuracy of ICD codes in Korea. Park et al. 4) re- 
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ported an 83% accuracy for a cerebrovascular diseases diagnosis 
based on ICD codes in 425 cases among Korean men in 2000. In 
the National Diagnosis Related Group Validation Study, the accura- 
cy of mixed diseases was 79.2%. 13) In the diabetes epidemiology re- 
port in Korea, the accuracy of admitted cases was 87.2%, and that 
of outpatients was 72.3°/o. 191 These results for other diseases seem 
to be comparable with our AMI accuracy results. It has been sug- 
gested that the accuracy of various mixtures of many diagnoses 
cannot be summarized or generalized due to the unique clinical 
situations and specific conditions of each disease. 201211 In an evalua- 
tion of the appropriateness of AMI diagnostic coding in Boston, 
41.7% of the cases failed to qualify in teaching hospitals because of 
inappropriate coding results, whereas the number was only 9.1% in 
non-teaching facilities. 221 Many of our participants were from teach- 
ing hospitals, and the coding results were acceptable. However, the 
differences in qualification status among hospitals may not be com- 
parable among countries due to huge variations in medical care sys- 
tems. 

In this study, we used WHO and ESC/ACC criteria to assess the 
AMI diagnoses. ESC/ACC criteria were introduced in 2000 and in- 
clude creatine kinase MB. 81 The WHO criteria, which have also been 
widely used, was the core method in the Multinational Monitoring 
of Trends and Determinants in Cardiovascular Disease project. 9 ' 101 

Several limitations of this study need to be acknowledged. We 
could not collect any information regarding the reasons for coding 
errors or on the various coding pathways in multiple hospitals. For 
example, the possibility of accidental AMI coding rather than sud- 
den death cannot be excluded. Questionnaires were sent instead of 
a direct review of the medical records. The number of cardiologists 
or staff who directly input the ICD codes was small, and the num- 
ber of no responses was large. Many of the participant hospitals 
were located in city areas and were teaching hospitals, so we could 
not expand our results to all hospitals or clinics. We evaluated all of 
the AMI cases using our approach, but they were not recruited ran- 
domly. We had a relatively small number of subjects, particularly of 
women (n=14, 13.1% of the 107 AMI cases). The reliability of the 
AMI diagnosis was fair to good regardless of these potential differ- 
ences among our samples. However, we cannot exclude the possi- 
bility of selection bias, because the participants were spontaneous 
examinees of health examination centers, although data from sim- 
ilar health examination centers has been used in previous cardiol- 
ogy studies. 23 ' 24 ' 

In conclusion, the accuracy of the AMI diagnosis using ICD-10 
codes in Korean MIC data was >70%, and reliability was good to 
fair. An age of 50 years or older was the only factor related to inac- 
curacy of AMI diagnoses; therefore, more attention is required when 
re-coding ICD data of older patients. Further studies should be 
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performed with regard to the validity of AMI MIC data and other 
cardiovascular diseases using a larger group. 
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